The Chaotic Nature of Faster Gradient Descent Methods

The steepest descent method for large linear systems is well-known to often converge very slowly, with the number of iterations required being about the same as that obtained by utilizing a gradient descent method with the best constant step size and growing proportionally to the condition number. Faster gradient descent methods must occasionally resort to significantly larger step sizes, which in turn yields a rather non-monotone decrease pattern in the residual vector norm. We show that such faster gradient descent methods in fact generate chaotic dynamical systems for the normalized residual vectors. Very little is required to generate chaos here: simply damping steepest descent by a constant factor close to 1 will do. Several variants of the family of faster gradient descent methods are investigated, both experimentally and analytically. The fastest practical methods of this family in general appear to be the known, chaotic, two-step ones. Our results also highlight the need of better theory for existing faster gradient descent methods. 1 Faster gradient descent methods Many efforts have been devoted in the two decades that have passed since the pioneering paper of Barzilai & Borwein [4] to the design, analysis, extension and application of faster gradient descent methods for function minimization; see, e.g., [13, 26, 6, 12] and references therein. These are methods that converge significantly faster than the method of steepest descent although, unlike the conjugate gradients (CG) method, they confine their search directions to the gradient vector at each iteration. ∗Department of Computer Science, University of British Columbia, Canada, (kvdoel/[email protected]), supported in part by NSERC Discovery Grant 84306.